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(57) In a feature extraction system for analyzing im- 
age data of an input document image, a feature extrac- 
tion method identifies image data as a table innage and 
identifies text image within the table image by perform- 
ing the steps of inputting image data of a document 
page, performing block selection processing on the doc- 
ument page, the block selection process identifies and 
separates the image data into bbcks having the same 
image type, identifying table image data based on the 
separated blocks of image data, identifying text blocks 
within the table image data, horizontally sorting all text 
blocks located in the table image data based on hori- 
zontal position information, vertically sorting all text 
blocks located in the table image data based on vertical 
positron information, separating text blocks into rows 
and columns based on a result of the vertical and the 
horizontal sorting steps, assigning column and row ad- 
dress coordinates to each text block in the table image 
data based on the separating step, and storing the as- 
signed address of each text block. 
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Description 

Ttie present inventicMi relates to a feature extraction system for analyzing a document page in order to distinguish 
between different types of image data in the document page utilizing block selection processing and for extracting 

s Image data within the document page for further processing based on the analyzed image data, and, more particularly 
the present invention relates to a feature extraction system which, upon identifying image data as a table, defines 
columns and rows in the image data identified as the table, assigns a row and column address coordinates to text 
blocks in the table, sub-divides certain text blocks into smaller text blocks, if necessary, and stores the coordinates in 
a hierarchical tree structure for post -processing extraction of the text data from the table using the column and row 

10 address coordinates. 

In conventional feature extraction systems, different types of image data within a document are first identified in 
accordance with image type and then grouped together based on type. Grouped innage data is subsequently extracted 
for further processing. Specifically, the feature extraction system executes a block selection routine so as to identify 
image data within a document, such as text data, table data, or graph data. Identified image data is grouped together 

IS based on image type, preferably, in a block rectangular format. Once the image data is grouped together in the block 
format, the innage data within the block is extracted for further processing based on its image type. 

Recently developed block selection techniques, such as the techniques described in EP-A-0567344 (US Serial 
No. 07/873,012) and EP-A-0712088 (US Serial No. 08/388,781) are used in feature extraction systems to provide 
automatic analysis of innages within a document page in order to distinguish between different types of image data 

20 within the document page. The result of such a bkxk selection technique is used to determine the type of subsequent 
processing to be performed on the image, such as optk:al character recognition (OCR), data compression, data routing, 
data extraction, etc. For example, image data which is designated as text data will be subjected to OCR processing, 
whereas image data which is designated as picture data would not be designated to OCR processing. As a result, 
different types of inr»ge data can be input and automatically processed without an operator's inten^entran. 

2S An example of how a block selection technique operates, such as the ones referenced above, will be discussed 

below with respect to Figures 1-3. 

Figure 1 shows the page of a representative document. Document page 1 is arranged in a two<;olumn format. 
Page 1 includes title 2, horizontal line 3, text areas 4, 5 and 6, whbh include lines of text data, halftone picture area 
7, framed area 8, and table 1 0. According to the block selectbn techniques described in U.S. Patent Application Serial 

30 Nos. 07/873,012 and 08/338,781 , each area of document page 1 is designated in accordance with the type of image 
data kx^ated therein and image data is then segmented based on its respective type. As the block selection application 
processes a document page, a hierarchical tree structure is created as shown in Figure 2. 

In Figure 2, hierarchical tree structure 20 contains a plurality of nodes which represent segmented blocks of image 
data of document page 1 . Each node of the tree contains feature data which defines the feature of each block of image 

3S data in the processed document page. For example, the feature data may include block location data, size data, attribute 
data (innage type such as text, picture, table, etc.), sub-attribute data, and chiki node and parent node pointers. In this 
regard, child or "descendent" nodes represent innage data which exist entirely within a larger block of image data. For 
example, text blocks located in a table would be designated as child nodes of the table block. Child nodes are depicted 
in the hierarchical tree structure as a node branching from a parent node, such as the nodes at the same level as node 

40 22, which branch from parent or root node 21 . 

Once a hierarchical tree structure, such as hierarchk;al tree structure 20, has been created, it is stored in memory 
Upon receiving a request to process the image data in the document image, hierarchical tree structure 20 is retrieved 
from memory and image data which has been blocked together is processed according to the feature data stored in 
its corresponding node. For example, a user can extract data within various text blocks of a table in order to further 

45 process the data. 

In addition to using the hierarchical tree structure to process image data of a document page, the hierarchical tree 
structure is used to generate and display a comprehensible format of document page 1 , such as the display of block 
template 40 shown in Figure 3. Block template 40 of the document page is generated and displayed to the user based 
on the feature data stored in the hierarchical tree structure shown in Figure 2. 

50 The block template of the document page directly reflects the structure of the hierarchical tree as well as the feature 

data stored in each node of the hierarchical tree structure. The feature data in the node is utilized to generate the block 
outline, location, and size. In addition, feature data in each node is used to identify the type of innage data contained 
within the block. For example, as shown in Figure 3, block template 40 includes text bkx;ks 41, 42 and 43, each of 
whk;h corresponds to nodes 28, 31 and 32, respectively. Also shown is table block 48 which includes text cells 49 

55 whk;h correspond to nodes 39A-39N, respectively. 

However, in the foregoing block selection techniques, difficulties have been encountered. For example, erroneous 
block selection processing of image data identified as a table occurs when text data within the table is grouped sepa- 
rately or grouped together due to the errors in block selection techniques or due to poor scanning or printing of the 
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analyzed document page. This results in poor post-processing of the table image data and text data therein. That is, 
when image data is identified as a table, the table includes text data which must be grouped together or separately 
within the table based on its column and row position in the table. This creates problems when text data is horizontally 
or vertically grouped and blocked with other text data which should not be Included in the block or when text data 
s should be horizontally or verticaify grouped and bkxked with other text data in the table. 

The present invention addresses the foregoing difficulties by providing a feature extractbn system which extracts 
inriage data within a table by defining columns and rows in image data identified as a table, assigning a row and column 
address to text data to be extracted from the table, and extracting text data from the table based on its column and 
row address. 

10 According to one aspect of the inventton. In a feature extraction system for analyzing image data of an Input 

document image, a feature extraction method identifies image data as a table image and Identifies text within the table 
image by performing the steps of inputting Image data of a document page, performing block selection processing on 
the document page, the block selection processing Identifying and separating the image data into blocks having the 
same image type, identifying table image data based on the separated blocks of Image data, identifying text blocks 

15 within the table image data, horizontally sorting all text blocks located In the table image data based on horizontal 
positk>n information, vertically sorting all text bkx:ks located in the table image data based on vertical position infor- 
mation, separating text blocks Into rows and columns based on a result of the vertical and the horizontal sorting steps, 
assigning column and row address coordinates to each text block in the table image data based on the separating 
step, and storing the assigned address of each text block. 

20 According to another aspect of the present Invention, a feature extraction method identifies table image data and 

locates text image data within the table by performing the steps of Inputting innage data of a document page, performing 
block selection processing so as to Identify and separate image data into blocks having the same image type, identifying 
blocks of image data which contain table image data, locating text blocks wrthin the table Image data, horizontally 
sorting all text blocks located in the table image data based on horizontal position information, vertically sorting all text 

25 blocks kx^ted In the table image data based on vertical position informatbn, separating text blocks Into rows and 
columns based on a result of the vertical and the horizontal sorting steps, assigning address information to each text 
block in the table Image data based on the separating step, and storing, in a node representing the block of table Image 
data in the hierarchical tree, table address Informatbn and storing, in respective nodes representing text blocks branch- 
ing from the node representing the table innage data, text block address information. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a representational view of a document page; 

Figure 2 Is a representational view of a hierarchical tree structure resulting from block selection processing on the 
35 document page Illustrated In Figure 1 ; 

Figure 3 is a view showing how the document page which Is created based on the hierarchical tree structure shown 
in Figure 2 might be displayed on a display screen; 

Figure 4 Is a perspective view showing the outward appearance of an apparatus according to the present invention; 
Figure 5 is a block diagram of the Figure 1 apparatus; 
40 Figures 6Aand 6B comprise a flowchart showing the methodfor performing table analysis and manipulation utilizing 

the bbck selection technique of the present inventbn; 

Figure 6C is a flow diagram for showing the detailed processing that was referred briefly to in steps S606-S609; 
Figure 6D is a flow diagram for showing the detailed processing that was referred briefly to in steps S610-S61 2; 
Figure 7 Illustrates an example of a basic table structure; 
45 Figures 8A and 88 are representational views of a group of text blocks which can be split into two form columns 

in a table; 

Figure 9A is a representational view of a table before undergoing column and row analysis processing according 
to the present invention and Figure 98 is a representatbnal view of that same table after column and row analysis; 
Figures 1 0A and 1 0B are representational views of a group of text blocks which have column address ranges and 
50 which can be sub-divided into separate columns; and 

Figure 11 A is a representational view of a hierarchical tree structure resulting from block selection processing on 
the table shown in Figure 9A and Figure 118 is a representational view of the same hierarchical tree after It has 
been updated when the table in Figure 9B has undergone column and row analysis. 

5S DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Figure 4 Is a view showing the outward appearance of representative computing equipment which incorporates a 
block selection based table manipulation and text data extraction system according to the present invention. 
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Shown in Figure 4 is cx>mputing equipment 410, such as an IBM PC or PC compatible computer having a windowing 
operation system such as Microsoft Windows Operating System. Computing equipment 410 is provided with either a 
monochromatic or color display monitor 412, on which computing equipment 410 displays images to the user. Com- 
puting equipment 410 is further provided with fixed disk drive 411 for storing data files and application program files, 
keyboard 413 for permitting input text data and nr^anipulation of objects displayed on display screen 412, and pointing 
device 414, such as a mouse, which is provided by pointing and for manipulating objects displayed on display screen 
412. 

A multi-page document is input by scanner 416 which scans each page of the document or other images and 
provides a bitmap image data of those pages to computing equipment 410. The image data is stored into disk 411 in 
either compressed or uncompressed format. 

A conventional color printer 418 is provided for outputting documents processed by computing equipment 410. 

Also provided are network interface 424 for interfacing with a local area network, and facsimileAnodem interface 
426 for sending and receiving facsimile messages and other data files via a facsimile/modem. 

In accordance with operator instructions, and under control of the windowing operating system, stored application 
programs, such as desktop publishing programs, drawing application programs, and block selection applications, are 
selectabty activated to process and to n^nipulate data. Also in accordance with operator instructions and based on 
those stored application programs, commands are issued to display images on monitor 41 2 and to print images ap- 
pearing on monitor 41 2 by printer 418. 

Most typically, the applicatk>ns stored on computer disk 411 , including the block selection program of the present 
invention, have been stored on disk 41 1 by downloading each application from a computer-readable medium, such as 
a floppy disk, a CD-ROM or a computer bulletin board. 

Figure 5 is a detailed block diagram showing the intemal construction of computing equipment 410. As shown in 
Figure 5, computing equipment 410 includes a central processing unit (CPU) 520, such as a programmed microproc- 
essor, interfaced to computer bus 521. Also interfaced to computer bus 521 is scanner interface 522, printer interface 
523, network interface 524, fax modem interface 526, display interface 527, keyboard interface 528 and mouse interface 
529, 

Main memory 530, such as random access menrrary (RAM), interfaces to computer bus 521 so as to provkJe CPU 
520 with access to memory storage. In particular, when executing stored application program instmction sequences, 
such as those associated with application programs stored on disk 411, CPU 520 loads those instruction sequences 
from disk 41 1 (or other storage media such as media accessed via network interface 524 or via a floppy disk drive (not 
shown)) into nnain memory 530 and executes those stored program instruction sequences out of main memory 530. 

[Table And Text Bkx:k Identification Using Block Selection] 

Briefly, the present invention provides a computer-implemented method which tocates and identifies table cells of 
text in each text block within image data identified as a table and determines whether that text block can be sub<livided 
into one or more additional columns and rows depending on the location of the table cells of text in the text block. 

According to the present invention, once a table has been located in image data, text blocks in the irmge data are 
identified using known-block selection techniques such as those discussed previously. Upon identifying text blocks, 
the text blocks are sorted based on horizontal and vertical positbnal information within the document page and then 
assigned address coordinates based on row and column location in the table. Once assigned address coordinates, 
each text block is analyzed again to locate text blocks having an address range, such as a column address range, a 
row address range, or both. Text blocks having an address range are subjected either to vertical or horizontal separation 
processing based on the type of address range (for example, row address range, receives vertical separation process- 
ing) in order to determine if the text block can be sub-divided into smaller text blocks by inserting column and row 
separators between table cells of text in the text block. If sub-divided, each text block is assigned address coordinates 
within the table based on the newly positioned column and row separators. The address coordinates of all text blocks 
are then stored in order to update the hierarchical tree structure in accordance with known block selection techniques. 

Figures 6A-6B comprise a flowchart showing the method for performing table identification, and text block sub- 
division and text block addressing using the block selection application according to the present invention. The process 
steps shown in Figure 6A-6B are executed by CPU 520 in accordance with a computer process steps stored in disk 41 1 , 

In step S600, pixel image data of a document page is input into the system and then stored in disk 411. Image 
data is a pixel-by-pixel representation of the image, preferably the pixel data is binary pixel data, that is, black and 
white image data which has been input through scanner 416, through network interface 424 or through facsimile/ 
modem interface 426. In step S601 , the block selection application is launched. For the sake ot brevity, a description 
of block selection initialization and processing of an entire document page of different image types will not be provided 
herein. Rather, a description of how block selection and analysis of image data identifies a table will be described. For 
a detailed description of how block selection techniques distinguish between image data within an entire document 
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page, block selection techniques disclosed in Serial Nos. 07/873,01 2 and 08/338,781 should be reviewed. The present 
invention is directed to identifying image data as a table and to sub-dividing, if necessary, text blocks which span more 
than columns and/or rows of the table into smaller text blocks and within the table once the block selection process 
has sub-divided a text block. 

5 

[Table Identification] 

A brief description will now be provided as to how block selection identifies a blocked area as a table in step S603. 
For a more detailed description of this processing, the block selection technique disclosed in Serial No. 07/873,012 
10 should be reviewed. 

Thus, in step S603, when framed data is located using one of the block selection techniques discussed above, it 
is analyzed to determine if the frame is a Irame-table' or a "frame-halftone". Accordingly, the frame data is further 
processed to determine if it includes table or tabularly organized data by inspecting the interbr of connected compo- 
nents to obtain white contours which are white pixels that are inspected rather than black pixels. In this regard, the 

^5 interior of a non-text unit is scanned in a first direction from a bottom right-hand portion of an interior of the non-text 
unit toward an upper left-hand portion. When a first white pixel is encountered, adjacent pixels to the white pixel analyzed 
are in an order similar to a starburst-like pattern. Accordingly, white contour tracing according to the starburst-like 
pattern is performed in a four directbn vector White contour tracing is continued in the four directions until all white 
contours are encbsed by black pixels or traced. After each white contour is located, scanning proceeds as discussed 

20 above in the first direction until all white contours enclosed in a nontext object have been traced. 

At this point, a density of the non-text unit is calculated by counting the number of black pixels within the connected 
component, and dividing the number of block pixels by the total number of pixels enclosed by the rectangular The 
number of white contours found within the non-text unit are inspected. If the number of white contours is greater than 
or equal to a predefined number, then there is a possibility that the non-text images, in fact, are a table or tabularly 

2S arranged series of text blocks. 

Next, the white contourfilling rate is determined. The white contour filling rate is the degree to which white contours 
fill the area enclosed by the non-text image. If the filling rate is high, then it is probable that the non-text image is a 
table or tabulated arranged sequence of text data. However, If the filling rate is low, then it is most likely not a table or 
tabularly arranged sequence of text data. 

30 If the filling rate is high, then it Is probable that non-text Image is a table or tabulated arranged sequence of text 

data. To increase the confidence in this determination, the white contours are inspected to determine if they form a 
grid-like structure extending both horizontally and vertically. Specifically, non-grid arranged white contours get recom- 
bined if their boundaries do not extend horizontally and vertically across at least two contours. White contours for a 
table are not likely to recombine, whereas white contours for a non-table, for example, a halftone image or a line 

3S drawings, are more likely to recombine. 

Accordingly, the recombination rate is inspected. 

If the recombination rate is high or if the number of white contours rennaining after recombination is greater than 
a predetermined number, then the non-text unit is designated as a halftone image or a line drawings. However, if the 
recombination rate is less than the predetermined number, then the non-text image is designated as a "table". 

40 

[Text Bbck Identification, Sub-Division And Addressing] 

Now, once the frame area has been designated as a table, in step S605 text blocks within the area designated as 
a table are located. That is, the interior of the newly designated table area is inspected so as to detect and to classify 

45 connected components in eight directions. The process for locating text blocks in a table is described further in U.S. 
Patent Applicatkxi Serial No. 07/873,01 2. 

From the block selection process, the identified table may contain a plurality of text blocks inskle the table. However, 
the text blocks are randomly processed and are assigned position information based on a location within the document 
page. As a result, the text blocks are not placed in a column/row order within the table, but rather branch randomly 

50 from the table node when stored in the hierarchical tree structure. This means that for each of the text blocks within 
the table, it is possible that other text blocks can be located within the same isolated white area, which is due to editing 
style, or poor printing or scanning of the document page. Accordingly, text blocks may contain more than one table 
cell of text in either the horizontal or the vertical direction or both. That is. after bkx;k selection, each text block within 
the table should correspond to only one table cell. However, as stated above, one text block may contain more than 

55 one table cell content in either the horizontal or vertical direction or both. /Vnd, one table cell content may be split into 
more than one text block. 

In order to properly extract specific text data from the table for post^jrocessing purposes, the present invention 
analyzes the table cell structure and assigns column and row address coordinates to text blocks and then analyzes 
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text blocks in order to determine if a text block containing one or more table cells can be further divided into smaller 
text blocks. 

Thus, in step S606, analysis of the table structure is performed. However, prior to analyzing the table structure, a 
description of a table will be defined. A basic table structure is defined as shown in Figure 7. As shown In Figure 7, 

5 table 600 is defined by five grid lines, which make up fou r columns and three rows. Each text block is assigned positional 
information from the block selection processing. The positional lnformatk)n relates to the text block location in the 
analyzed document page rather than a position relative to the table. In addition, as shown in Figure 7, table 600 Includes 
multiple text lines in rows 2 and 3, Accordingly, without the present invention, in order to extract and post-process only 
one line of the multi-line row, tor example, table cell 601 , the entire table cell or text block would have to be extracted. 

10 However, after applying the method of the present invention, table cell 601 as well as the remaining text lines in rows 
2 and 3 would be further sub-divided into Individual rows or text blocks, thereby permitting individual lines within a row 
to be extracted. 

Thus, in step S606, all text blocks In the identified table are sorted based on horizontal positional Information. For 
example, all text blocks residing In the first column are grouped together. To accomplish this task, a block having the 
IS furthest left coordinate is selected as being In group 1 . Then, all other text blocks are compared to the group currently 
being formed using a maximum horizontal overlapping method. That is, a aiaxlmum horizontal overlapping rate of the 
currently formed group and the selected text block are compared to a predetermined threshold t_hV The maximum 
overlapping rate is detemnined by the following equation; 

20 max_h_overlapplng = max (horizontal overlapping area/width of currently formed group, horizontal overlapping ar- 
ea/width of selected text block) 

If nnax_h_overlapping is greater than *t_h^, then the selected text block is placed in a currently fomied group and 
the group horizontal coverage Is updated. If no overlapping occurs or the maximum overlapping Is less than "t.h", then 

25 the selected text bkx;k is placed into a separate group. All text blocks in the table are analyzed using the same method. 

Once all the text blocks have been sorted into similar groups according to horizontal location, in step S606, a group 
of text bkx:ks are sorted In order to determine sequential order of the group In the table. This Is performed by analyzing 
the horizontal location informatbn (step S608). 

Based on the sorting performed in step S608, in step S609, columns for the table are calculated based on sepa- 

30 ratrons or gaps Inside each text block group. Specifically, the text blocks Inside each group are assumed to belong to 
one column area, based on positbnal informatkxi, and are determined to be in one column of the table. However, there 
are situations, for example, as shown in Figures 8A and 8B, In which text blocks in the same group belong to more 
than one column area and cannot be easily separated since they span one or more columns. 

In such Instances, the block selection process of the present Inventbn locates these text bkxks which overlap a 

3S column area in order to determine where the text block begins and ends. In order to accomplish this task, it is determined 
where a column or columns can intersect the text block which spans more than one column area. Based on this de- 
termination, a column range can be determined. Once each text block which overlaps a column area is located and 
analyzed, each text block is assigned a column address range based on the relative location between the block and 
the adjacent groups on either side. Similarly, the remaining non-spanning text blocks are assigned a column address. 

40 For example, as shown in Figure 8A, text bkxk 701 spans a gap between two columns of neighboring text blocks within 
the same group, and, accordingly, a column address range is assigned to Is text block 701. That is, text block 701 
assigned (n,20 a column address range of n+1 ), wherein n-1 is the last column number of the previous group. 

How to sort text blocks (S606) and groups of text blocks (S608) based on horizontal location and how to calculate 
and assign column addresses (S609) is explained In greater detail in connection with Figure 6C, described below. 

45 Figures 6C is a flow diagram for explaining In greater detail the process performed in steps S608-S609. 

Referring to Figure 6C, after block selection is performed on the table structure and text blocks are defined in the 
table, the coordinates of each of the text blocks are utilized in step S630 to sort all text blocks according to their 
horizontal position. Specifically, the left-hand coordinate of each text block in the table Is examined to determine which 
block is the furthest left within the table. The text block having the furthest left location is placed at the beginning of a 

50 sort list. Each of the text blocks thereafter are analyzed based on their left-hand coordinate to determine its place within 
the sort list. For example, a text block having coordinates of (1 ,1 ) will be placed at the beginning of the list whereas a 
text bkx;k having the coordinates of (3,5) will be placed at the middle or end of a sort list. Accordingly, each text btock 
is sorted based on their horizontal location. 

In step S631 , the first text block in the sorted list is retrieved and is placed in the first group (column). Utilizing the 

ss coordinates of the first text block in step S632, the horizontal coverage of the first group Is initialized as the first block 
and used to begin a comparison process between the remaining text blocks. In step S634, it Is determined if there are 
other text blocks within the table and, if so, in step S635, the coordinates of the next text block are retrieved and 
compared to the horizontal coverage of the group currently being formed in order to detemnlne whether the next text 
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block resides in the same group as the last text block. In order to perform the comparison, a maximum horizontal 
overlapping rate of the current group and the next text block are examined. 

If the maximum overlap is greater than a predetermined threshold "t.h", then the right-hand horizontal coverage 
or coordinate is reset based on the right hand coordinate of the next text bkx:k whch is analyzed in step S636. That 
s is, the right-hand coordinate of the next text block which was analyzed can be used to determine a new horizontal 
coverage of the group. On the other hand, if the maximum overlap is not greater than the predetermined threshold "f 
in step S636, flow returns to step S632 and the next text block begins a new group and a new group horizontal coverage 
is initialed in step S632. 

Reverting back to step S634, in the case that no additional text blocks remain in the table, flow proceeds to step 

10 S640 in which the groups of text blocks are ordered horizontally. In step S641 , if any overlapped groups exist, they are 
combined. Specifically, in the case that neighboring groups horizontally overlap to a predefined threshold ■x_h", these 
groups are combined Into one group. In step S642, column separators are placed between groups and inside some 
groups, if necessary, based on text block assignments. Specifically in step S642, the width and length of a gap is 
examined. That is, in the case that the wkfth meets a predefined criteria "w" and the length of the gap meets a predefined 

IS criteria T, then it is determined that a column separator can be inserted into a group. On the other hand, if the gap 
does not meet one or both of the width and length criteria, then no column separator is inserted inside the group. In 
step S645, column addresses are assigned to text blocks based on the column separators. In this regard, column 
separators are put between groups by default, and column separators may be put inside some groups in which column 
calculation (step 3609) determines that a group contains more than one column span. 

20 Reverting back to Figure 6A, in step SB 10 once all possible columns are calculated in step 8609, all text blocks 

are sorted into groups of rows according to a vertical positional information. Row analysis is performed similar to column 
analysis performed in steps S606-S609. Specifically, in step S610, all text blocks are sorted into groups according to 
vertica! positional infornretion. Once again, text blocks are placed into different groups In which each text block has 
some vertical overlapping with another text bkx;k of the same group which is either closely to the right or to the left of 

25 it. As stated previously, the maximum vertical overlapping rate of the text block and the group currently formed is 
compared to some predetermined threshold "t_v*. 

max_v_over lapping = max(vertical overlapping area/length of currently formed group, vertk:al overlapping area/ 
length of text block selected) 

30 

In step S611 , the groups of text blocks are then sorted based on the vertical coverage of each group. Row sepa- 
rators are calculated for each group, and once again, text blocks inside each group are assumed to belong to one row 
area and each text block therein is assigned a row address in step 8612. Once again, text blocks overlapping a row 
separator is assigned a row address range. 
35 How to sort text blocks (S61 0) and groups of text bkxks (S611 ) according to vertical location and how to calculate 

and assign new addresses (8612) is explained in greater detail in connection with Figure 6D, described below. 

Figure 6D is a flow diagram for explaining in greater detail the process defined in steps S610-S612. 

Referring to Figure 6D, after column analysis, the text block coordinates are used for row analysis. Specifically in 
step 8650, row analysis begins by examining the top coordinate to determine which bkx:k is the closest to the top of 
40 the table. The text bkx:k whk:h is at the top kxation of the table is put at the beginning of the sort list. Each of the text 
blocks thereafter are analyzed based on their top coordinate to determine its place within the sorting list. For example, 
a text block having coordinates of (1,1) will be placed at the beginning of the list whereas a text block having the 
coordinates of (3,5) will be placed at the mddle or end of a sort list. Accordingly, each text block is sorted based on 
their vertical location. 

45 In step 8651 . the first text block in the sorted list is retrieved and is determined to be in the first group (row). Utilizing 

the coordinates of the first text block in step S652, the vertical coverage of the first group is initialized as the first block 
and used to begin a comparison process between the remaining text blocks in the table. In step S654, it is determined 
if there are other text blocks within the table and, if so, in step 8655, the coordinates of the next text block are retrieved 
and compared to the vertical coverage of the currently formed group in order to detemnine whether the next text block 

50 resides in the same group as the last text block. In order to perform the comparison, a maximum vertical overlapping 
between the currently formed group and the next text block are examined. 

If the maximum overlap is greater than a predetermined threshold "t_v*, then the vertical coverage or bottom 
coordinate may be reset based on the next text bkx:k which is analyzed in step S656. That is, the bottom coordinate 
of the next text block which was analyzed can be used to detenmine a new vertical coverage of the group. On the other 

5S hand, if the maximum overlap is not greater than the predetermined threshold ■t_v' in step S656, flow returns to step 
S652 and the next text block begins a new group and a new group vertical coverage is initialized as the next text block 
in step 8652. 

Reverting back to step 8654, in the case that no additional text blocks remain in the table, flow proceeds to step 
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S660 in which the groups o1 text blocks are ordered vertically. In step S661 , if any overlapped groups exist, they are 
combined. Specifically, in the case that neighboring groups vertically overlap to a predefined threshold "x_v", these 
groups are combined into a single group. In step S662, row separators are placed between groups and inside sonne 
groups, if necessary, based on the assigned groups. In step S665, once the row calculations of each group has been 
s performed, row address assignments are provided in accordance with the number of row separators between groups. 

After column and row analysis, each of the text blocks have been assigned a column and row address coordinates. 
For example, in table 800, block 801 is assigned row/column address (1,1), block 802 is assigned row address and 
column address and address range (1, 2-4), block 803 is assigned (2,1), block 804 is assigned row/column address 
(2,2). block 805 is assigned row/column address (2,3), and block 806 is assigned row/column address (2,4). In addition, 
10 large text block 808 is assigned both a row and column address range (4-7, 2-4). 

Once the text block colu mn and address coordinates have been assigned, in step S61 5, the block selection process 
of the present inventk>n locates text blocks which span gaps and contain either a column address range or a row 
address range. These text blocks are analyzed again to determine if they can be sub-divided by using the column and 
row separation process to determine if column or row separators can be inserted within the range address of the text 
IS block. Specifically, in step S61 7, it is determined if there is a large horizontal gap around a column separator, as shown 
in Figure 10A or if there is an incomplete vertical grid line around a column separator as shown In Figure 10B. The 
same procedure is then performed for row separators. 

Thus, in step S617, it is determined if a separator can be inserted into the text block having an address range. If 
row or column separators can be applied in step S618, the text block Is divided into smaller text blocks. That is, the 
20 text block itself is divided into at least two columns or rows. Each new text bkx:k is then assigned an address after the 
column separator or row separator has been applied. In the example shown in Figure 8 A, text block 701 in group 700 
spans a narrow gap within the group. Using the column or row separator method, the width and length of the gap is 
analyzed to determine if column separator 710 can be placed therein. If column separator 710 can not be placed therein 
since the gap width does not meet a predefined criterion, the text block can not be sub<iivided by a column separator. 
2S In Figure 8A, column separators 711 and 712 indicate columns which separates group 700 from another group. Using 
these separators as boundaries to a column, each text block in group 700 is given a column address assignment. 

As shown in Figure 8B, there is another example in which there is a gap existing between two vertical lines 720 
in group 716 of text blocks. In this situation, after the gap is analyzed it is determined that the gap has a certain width 
and length which is equal to or greater than a predefined criteria. Accordingly, each text block which has a column 
00 address range of (Y-Y+1 ), where Y-1 is the last column number of the previous group, is separated into two smaller 
text blocks since in each block a wide gap exists around the column separators. That is, column separator 722 shown 
as a dotted line is inserted to separate each text bkx;k into two snr^ller text blocks. Thereafter, each text block is 
assigned a new column address based on the sub-division. 

In Figure 9 A, text bkx;k 808 has both a column and range address and also contains more than one table cell. 
35 Accordingly, this type of text block is a good candidate for further sub<ii vision using the column and row processing. 
Thus, after applying column and row processing, text block 808 is sub-divided into 12 new text blocks as shown in 
Figure 9B. Each new text bkxk is then assigned an address based on its column and row in the newly divided table 800. 

In step S618, if the text block with a column address range has "n" column separators, then it will be determined 
that this text block actually contains, at most, (n+1 ) column contents. For example, as shown in Figures 1 0A and 1 08, 
-M> if a text block includes nrx>re than one column separator, then the column separators are used to cut the text block into 
different columns and smaller text blocks are formed. As shown in Figure 10A, there exists a large gap. In the example, 
this gap meets the width criterbn and, as a result, column separators 901 and 902 are inserted in order to form three 
columns. Accordingly, text block 903 is split into three text blocks based on column separators illustrated In Figure 
10A. The column address of each new text block is then addressed as column n, column n+1, and column n+2. 

Similarly, as shown in Figure 108, text block 913 includes an incomplete line 911 which exists near column sep- 
arator 912 as well. In this case, text block 913 is split into two text blocks using column separator 912. Each new text 
block is assigned a column address of column n and column n+1 . 

Once all text blocks with ranges have been divided into smaller text blocks, in step S61 9, text blocks which have 
the same column and row address are combined into one text block. This may occur when groups have been combined, 
so as discussed previously, or if a gap does not meet the predefined criteria. 

In step S620, column and row address coordinates of the table are stored in menrK>ry in order to update the hier- 
archical tree structure. Prior to updating the hierarchical tree structure for table 800, hierarchical tree structure 1100 
has a page node, a table branching from the page node, and 1 5 text block nodes branching from the table node. Once 
table 800 undergoes processing according to the present invention, each table cell of text is sub-divided into columns 
ss and rows and, utilizing the stored address coordinates, the hierarchk;al tree structure of the document page can be 
updated as shown in Figure 118. 

Figure 118 shows hierarchical tree structure 1100 after applying block selection processing of the present invention 
to table 800 shown in Figure 9 A. Hierarchical tree structure 1100 shows the hierarchy for table 800 shown In Figure 
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9B. It is to be understood that any other type of image data located in a page would be shown as well but those nodes 
are not shown for the purposes of brevity and clarity. Thus, hierarchial tree structure 1100 shows table node 1101. 
Branching off table node 1103 are text blocks 1-26, of which only textblockl, textblockS, textbkxiklO, textblockIS, 
textbk)ck20, textbk5ck25 and textblock26 are shown, 

s As discussed above, textblockl 5 contained a plurality of indivklual cells of text which have now been sub-divided 

into smaller text blocks when submitted to column/row separation processing. The result of processing the text block 
results in smaller text blocks which are assigned individual address coordinates. This result is shown in Figure 11 B as 
26 text blocks branching off table node 1103 in hierarchical tree structure 1100. 

As can be readily understood, the result of processing a table using the present invention results in a logically 

10 defined hierarchical tree structure of a table whk:h will be used for post-processing the table image data in the document 
page. For example, utilizing the column and row information of text blocks stored in hierarchical tree structure 1100, 
text data can be properly extracted based on specific column and row address coordinates. If some specified address 
coordinate is related to a text block with a single column and row address, then an internal image of the text block will 
be extracted. However, if some specified address of a text block includes either a column address range or row address 

IS range, then all of the cells of text within the text block is extracted. As shown in Figure 9, if text block 803, which is the 
same as text blocks 804 or 805, is requested, the whole content of the text block is extracted. However, if cell 808 
could not be further divided, then all cells of text inside block 808 are extracted. On the other hand, in the case of Figure 
9B, after performing step S61B, text block 808 is sub-divided into 12 smaller text blocks and each are sequentially 
addressed as (4,2) - (4,4), (5,2) - (5,4), (6,2) - (6,4), and (7,2) - (7,4). In this regard, it is noted that since text block 802 

20 does not include any gaps between text data, block 802 remains the same since no column or row separators could 
be inserted between the text data within the block. 

The Invention has been described with respect to a partrcular embodiment. It is to be understood that the present 
inventbn is not limited to the above description but may be applied to all types of image data and not just table image 
data. In additbn, various changes and modifications may be made by those of ordinary skill in the art without departing 

25 from the spirit and scope of the invention. 



Claims 

30 1 . I n a feature extraction system for analyzing image data of an input document image, a method for identifying image 
data as a table image and for identifying text within the table image, the method comprising the steps of: 

inputting image data of a document page; 

performing bkx:k selection processing on the document page, said block selection processing identifies and 
3S separates the image data into bkx:ks having the same image type; 

identifying table Image data based on the separated blocks of image data; 
identifying text bkx^ks within the table image data; 

horizontally sorting all text blocks located in the table Image data based on horizontal position information; 
vertically sorting all text btocks located in the table image data based on vertical position information; 
40 separating text blocks into rows and columns based on a result of the vertical and the horizontal sorting steps; 

assigning column and row address coordinates to each text block in the table image data based on the sep- 
arating step; and 

storing the assigned address of each text block 

45 2. The method according to Claim 1 , further comprising the step of analyzing all text blocks having a column or row 
address range In the table Image data In order to determine If the text block can be further separated wherein each 
text block having either a column or row address range is analyzed to determine If a text block having either column 
or row address range can be further divided Into at least one addltbnal column or row, and, if it is determined that 
a text block can be further divided, dividing the text block. 

so 

3. The method according to Claim 1 , further comprising the step of assigning a new text block address to text blocks 
divided in the analyzing step. 

4. The method according to Claim 1, further comprising the step of storing the new text block address for each text 
55 block which was assigned an address in the text block address assigning step. 

5. In a feature extraction system which creates a hierarchical tree structure corresponding to a block template of a 
document page, wherein the hierarchical tree structure includes a plurality of nodes each of which represents a 
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block of image data in the block template of the document page and contains address data defining locations of 
the image data in the document page, a method of Identifying table image data and locating text image data within 
the table, comprising the steps of: 

inputting image data of a document page; 

performing block selection processing so as to identify and separate image data into blocks having the same 
image type; 

identifying blocks of image data which contain table image data; 
locating text blocks within the table image data; 

horizontally sorting all text blocks located in the table image data based on horizontal position infomnation; 
vertically sorting all text bkx;ks located in the table image data based on vertical position information; 
separating text blocks into rows and columns based on a result of the vertical and the horizontal sorting steps; 
assigning address infomnatbn to each text block in the table image data based on the separating step; and 
storing, in a node representing the block of table image data in the hierarchical tree, table address informatton 
and storing, in respective nodes representing text blocks branching from the node representing the table image 
data, text block address informatbn. 

6. The method according to Claim 5, further comprising the step of analyzing all text blocks having a column or row 
address range in the table image data, wherein each text block having either a column or row address range is 
analyzed to determine if a text bkx;k having either column or row address range can be further divided into at least 
one additional column or row, and. If it is determined that a text block can be further divided, dividing the text block. 

7. The method according to Claim 5, further comprising the step of assigning a new text block address to new text 
blocks divided out in the analyzing step. 

8. The method according to Claim 5, further comprising the step of storing, in a node branching from a text block 
from which a new text bkx;k was divided out, address information for the new text block. 

9. A feature extraction system for analyzing image data of an input document image, comprising: 

input means for inputting image data of a document page; 

block selectk>n processing means for performing block selection processing on the document page, said block 
selection processing means identifying and separating the Image data into blocks having the same image type; 
identifying means for Identifying table image data based on the separated blocks of image data; 
identifying means for Identifying text blocks within the table image data; 

horizontal sorting means for sorting of all text blocks located in the table image data based on horizontal 
position informatbn; 

vertical sorting means for sorting of all text blocks located in the table image data based on vertical position 
Information; 

separating means for separating text blocks into rows and columns based on a result of the vertical and the 
horizontal sorting; 

address assigning means for assigning address information to each text block in the table image data sepa- 
rated by the separating means; and 

storing means for storing the assigned address of each text btock. 

10. The feature extraction system according to Claim 9, wherein the sorting means analyzes all text blocks having a 
column or row address range in the table image data in order to determine If the text block having either a column 
or row address range can be further divided into at least one additional column or row, and, If so, dividing the text 
block into at least two new text blocks. 

11. The feature extraction system according to Claim 9, wherein the address assigning means assigns a new text 
block address in the case a text block having either a column or row address range is further divided into at least 
two new text blocks. 

12. The feature extraction system according to Claim 9, wherein the storing means stores the text block address for 
each text block which was divided by the sorting means. 

13. The method according to Claim 1 , wherein, in said inputting step, the image data of the document page is input 
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by a scanner. 

The method according to Claim 5, wherein, in said inputting step, the image data of the document page Is input 
by a scanner. 

The feature extraction system according to Claim 9, wherein the input means is a scanner 
A method, apparatus or system as claimed in any combination of the preceding claims. 
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Canon Wants Mutually 
Rewarding Coexistence 

Source: Fortune. 7/29/91 

Ryuzaburo Kaku, Chairman of the Board of 
Canon, Inc. in his recent interview had the following lo say 
about the corporate world. The worid is divided into four 
types of companies: 

(1) Purely capiialisiic enterprises that exploit 
their workers for profit. 

(2) Those where management and labor work 
doscly together (o maximize profits, but don't pay enough 
attention to the community 

(3) A company thai both tries to make money 
but also seeks to fulfill its corporate responsibilities to 
society, but in a small scale way to a particular country or 
region. 

(4) A highly evolved type of company that 
contributes positively to world prosperity. 

Qmon is aspiring lo be the fourth type of com- 
pany. This is a company that is socially responsible and 
practices good corporate dlizenshtp at home and overseas 
and that can be referred to as a true global corporation. 
We have a basic philosophy to achieve a mutually reward- 
ing coexistence among employees, shareholders, cus- 
tomers and the communities in which we do business. 




Canon Develops World's First 
Ferroelectric Liquid Crystal Display 



Source: 



Wall Street Jovroa), 10/2/91, 
Canon PresK Release, 10/1/9] 



In a news confeicncc ycslcrdiy, Himshi Tanaka, a 
Caoon senior managing dircdor said that the company has suc- 
ceeded in developing the workTk fiisl ferroelectric liquid crystal 
(PLC) display screen. The screen will be teal marketed next 
spring m Oanon^ EZPS Japanese language DTP system. 
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Canon's Corporate Culture to Blend 
Best of U.S. & Japan 

Source: Fortune. 8/2f^l 

Mr. Hideharu Takemoto, President of Canon 
II.S.A. was recently interviewed by Fonune Magazine 
and had the following to say about Canon in North 
Anterica. 

Mr. Takemoto wants to create a new Canon 
corporate culture. The best of American and Japanese 
cultures must be blended to produce a richer corporate 
alchemy— a new ideal." 

Canon wants to create more jobs for 
Americans in the 1 990s and lo make them an integral 
part of the Canon family. Mr. Takemoto is further 
committed to cultivating local talent and moving local 
executives up through the ranks of Canon's highest cor- 
porate echelons. 
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